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NEURAL  NETWORKS 


1.  HISTORICAL  REVIEW 

A  construct  which  is  called  a  ’’neural  network”,  a  ’’neuromorphic 
system”,  a  ’’connectionist”,  or  an  ”organic-like  design”,  which  has  been  set 
aside  for  nearly  two  decades,  is  now  becoming  one  of  the  most  crucial 
emerging  technology  topics.  -  Tactical  Technology  Office  of  the  U.S.  Defense 
Advanced  Research  Projects  Agency  (DARPA7TTO)  labels  this  technology  ”as 
important  as  the  atom  bomb”.  DARPA  has  announced  a  proposed  $390 
million  neural  network  program  at  the  International  Conference  on  Neural 
Networks  in  San  Diego,  California,  July  1988  (1). 

Unlike  a  Von  Neumann  type  of  computer  which  needs  to  be 
programmed  to  carry  out  an  information-processing  function,  neural  networks 
are  promised  as  an  alternative  to  artificial  intelligence  (AI)  to  attack  many 
problems  AI  has  been  unable  to  do  such  as  seeing,  speaking,  reading,  and 
learning  from  experience.  While  conventional  digital  computers  process 
detailed  steps  of  a  stored  program  to  carry  out  its  goal,  neural  nets  can  be 
adaptive  or  trainable  through  a  series  of  trials  to  learn  how  to  process 

.  . — i4 

information.  i  -> 

The  philosophy  behind  neural  networks  is  to  t  •>  to  build  computers  that 
work  as  the  way  the  human  brain  does.  Researchers  Dwgan  to  devise  models 
for  the  structure  of  the  brain  in  the  1940s.  Frank  Rosenblatt,  a  scientist  at 
Cornell  University,  built  a  simple  neurocomputer  in  1957  with  eight  processing 
elements  (PE)  called  Perceptron  that  could  perform  typed-character 
recognition.  Professor  Stephan  Grossberg,  Boston  University,  made  significant 
progress  by  incorporating  mathematical  representation  with  neurobiology.  For 
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years,  computer  researchers  have  been  trying  to  create  machines  modeled  on 
the  human  brain  while  ignoring  the  complex  details  of  the  brain’s  anatomy. 
These  early  neural  networks  were  unable  to  handle  many  situations.  The 
research  was  abandoned  until  John  Hopfield  published  his  work  in  1982  and 
characterized  the  neural  networks  as  feedback  circuits  and  energy  functions. 

Today  with  better  understanding  about  how  the  human  brain  works,  the 
researchers  have  made  tremendous  progress  with  such  accomplishments  as 
multi-layer  systems,  advancements  in  mathematical  theories,  and  increased 
understanding  of  neurobiology  to  form  a  bridge  between  the  biologically- 
oriented  neural  modelers  and  the  artificial  neural  network  modelers.  The  first 
commercial  neurocomputer  was  developed  in  1985  by  TRW’s  AI  Research 
Center  in  San  Diego.  In  1986,  Terry  Sejnowsky  proved  that  a  neural  network 
could  learn  the  speech  patterns  of  a  6-year-old  child  (2).  Today  the  number  of 
firms  and  institutions  involved  in  some  form  of  neural  research  is  about  100, 
and  at  least  78  applications  have  been  developed  in  the  areas  such  as  vision, 
signal  processing,  classification,  robotics,  speech,  and  sensor  fusion  (3).  Even 
though  the  neural  network  is  still  in  its  infancy,  it  offers  the  prospect  of  saving 
substantial  amounts  of  time  and  human  effort.  However,  there  are  many 
factors  to  be  considered,  including: 

-  The  required  training  time  for  scaled  up  neural  networks  approaching 
the  capacity  of  the  human  brain. 

-  The  high  cost,  low  density,  and  low  speed  of  current  devices  as 
limitations  of  current  technologies. 
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2.  CURRENTLY  AVAILABLE 


2.1  Brain  and  Its  Applications  to  Neural  Networks 

2.1.1  Brain  Model 

The  human  cerebral  cortex  is  a  massive  parallel  processor,  consisting  of 
about  100  billion  neurons,  or  nerve  cells.  Each  neuron  has  roughly  1,000 
dendrites,  which  receive  electrochemical  impulses  from  sensory  organs  or 
other  neurons  by  means  of  a  specialized  contact  called  a  synapse.  The  human 
nervous  system  contains  about  100,000  billion  synapses.  If  the  system  operates 
at  100  Hz,  it  functions  at  about  10,000  trillion  interconnections  per  second. 
Each  nerve  cell  has  an  axon  that  sends  the  nerve  impulse  on  to  other  neurons. 

Current  psychological  theory  contends  that  logical  and  rational 
operations  are  located  on  the  left  hemisphere  of  the  brain,  while  creative  and 
intuitive  operations  are  on  the  right  hemisphere.  Neurocomputers  are  modeled 
on  the  way  the  right  hemisphere  processes. 

2.1.2  Neural  Networks 

A  neural  networc  consists  of  a  collection  of  processing  elements, 
corresponding  to  nerve  cells.  Each  processing  element  consists  of  several 
operational  amplifiers  and  resistors  with  a  small  amount  of  local  memory  and 
processing  power,  several  inputs  analogous  to  dendrites,  and  one  output 
anologous  to  a  neuron’s  axon.  A  variable  resistor  representing  the  synapse  is  a 
connection  between  the  output  and  input(s)  of  processing  elements.  When  a 
PE  receives  an  input  signal  that  exceeds  a  preset  threshold,  it  sends  an  impulse 
signal  to  others,  i.e.,  it  fires.  The  input  signal,  called  weight,  is  the  sum  of  its 
inputs  (dendrites).  The  set  of  weights  provides  a  self-adapting  dynamic  system 
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that  can  modify  its  response  to  each  input  pattern.  This  determines  the 
strength  of  the  connections  between  PEs  and  helps  the  system  identify  features 
encountered  in  the  past,  and,  therefore,  the  system  can  provide  the  correct 
output  for  each  input  applied  after  a  number  of  trials. 

How  to  build  a  neural  network  is  mainly  dependent  upon  how  the 
elements  are  connected.  There  are  several  possibilities  such  as  feed  forward, 
feedback  loops,  fully  connected,  sparsely  connected,  or  random  connections. 
The  nature  of  the  connections  depends  on  the  model,  or  architecture,  used  to 
construct  the  neural  network.  The  PEs  are  connected  into  several  layers.  One 
layer  receives  inputs  from  the  layer  above  and  provides  output  to  the  layer 
below  it.  Since  weighted  inputs  can  change  over  time,  the  network  can  adapt 
and  learn. 

There  are  three  basic  methods  used  to  train  a  neural  network  (3): 

-  Supervised  training:  This  training  requires  an  external  teacher  to 
provide  an  error  signal  when  the  network  provides  an  incorrect  response  in 
order  to  teach  the  network  the  correct  response  until  it  can  consistently 
produce  the  correct  response. 

-  Unsupervised  training:  This  training  requires  no  external  teacher. 

-  Self-supervised  training:  An  error  signal  is  generated  by  the  system 
until  it  produces  the  correct  response. 

2.2  NEURAL  NETWORKS  APPLICATIONS 

Neural  networks  vary  considerably.  At  least  50  different  types  and  78 
applications  have  been  developed  from  the  Perceptron  by  Rosenblatt  (1957)  to 
the  Boltzmann  and  Cauchy  machines  by  Hinton,  Sejnowsky,  and  Szu  (1986). 
The  number  of  processing  elements  range  from  eight  to  one  million,  the 


-  4  - 


number  of  connections  from  512  to  1.5  million,  and  the  number  of  connections 
per  second  from  one  thousand  to  several  million.  These  networks  exhibit  a 
variety  of  capabilities  from  typed-character  recognition  to  pattern  recognition 
for  images,  sonar  readouts,  and  radar. 

The  development  areas  and  applications  are  classified  as  follows  (3): 

-  Vision:  Image  classifier,  target  recognizer 

-  Speech:  Net  talk,  word  recognizer 

-  Classification:  Risk  analysis 

-  Signal  processing:  Sonar  classifier,  target  tracking,  adaptive  channel 
equalizer,  process  monitor 

-  Robotics:  Forklift 

-  Sensor  Fusion:  image  fusion 

And  the  following  applications  have  been  fielded: 

-  Risk  Analysis  System 

-  Process  Monito* 

-  Word  Recognizer 

-  Adaptive  Channe*  Equalizer 


2.3  Hardware/Performance 

2.3.1  Micro-electronics 

To  build  a  network  that  ha«  the  physical  characteristics  and  construction  like 
the  human  brain  is  impossible  with  current  technology.  However,  computer 
builders  are  looking  to  construct  devices  consisting  of  large  numbers  of  PEs 
that  behave  as  much  like  the  brain  as  possible. 
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Developers  including  Hecht-Nielsen  Neurocomputer  Corp.,  IBM  Corp., 
Science  Applications  International  Corp.,  Texas  Instruments  Corp.,  and  TRW 
Inc.  have  introduced  neurocomputers,  the  hardware  on  which  neural  networks 
can  be  implemented.  These  are  coprocessor  boards  that  plug  into  conventional 
machines  to  process  immense  quantities  of  information  (4). 

Carver  Mead’ s  artificial  retina  is  a  single  chip  containing  100,000  tran- 
sitors  designed  to  mimic  the  functions  of  a  biological  retina’s  layers  of  cells, 
with  10,000  interconnects  and  100  million  interconnects  per  second.  Although 
it  does  not  solve  ’’the  vision  problem”,  it  can  effectively  imitate  many 
functions  that  take  place  in  the  living  retina  (5). 

Several  other  technologies  which  are  expanding  hardware  capabilities 
beyond  their  current  limits  include  Gallium  Arsenide(GaAs)  and 
special-purpose  charge-coupled  devices  (CCD).  Multiprocessing  will  improve 
simulation  speed  by  an  order  of  magnitude. 

2.3.2  Optics 

Optics  is  the  least  mature  of  all  the  technologies  that  are  essential  to 
neural  network  research.  However,  it  will  be  an  alternative  when  other 
technologies  reach  their  ultimate  limits.  The  advantage  of  photons  over 
electrons  in  networks  is  that  the  number  of  channels  can  be  very  large  and 
close  together  without  interference  experienced  with  electrons,  and  the  light 
source  can  broadcast  to  millions  of  points.  The  optical  technology  can  offer  a 
very  high  interconnection  level  due  to  its  three-dimensional  nature.  The 
development  of  the  spatial  light  modulator  (SLM)  is  critical  to  neural 
networks.  SLM  as  a  transducer  converts  a  two-dimensional  pattern  of  light 
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into  a  spatial  pattern  that  can  vary  its  brightness.  SLM  can  perform  logic 
operations,  clocking,  and  can  be  viewed  as  a  matrix-vector  multiplier  (6). 


2.4  Application  to  IMA 

2.4.1  Information 

Even  though  neural  networks  can  be  applied  to  variety  of  tasks  and 
problems,  the  Defense  Department  has  been  the  principle  organization 
supporting  financially  the  research  and  development  for  this  field.  DARPA, 
through  its  Advanced  Stragic  Computing  Program,  is  funding  efforts  at  Texas 
Instruments,  AT&T,  IBM,  Bendix,  TRW,  and  General  Electric.  Most  studies, 
therefore,  concentrate  more  on  applications  related  to  national  defense.  These 
applications  involve  electronic  information  gathering  and  very  high  bandwidth 
sensors  in  order  to  help  process  information  in  real  time.  However,  these 
technologies  must  continue  to  mature  and  will  take  several  years  of 
implementation  and  improvement  before  being  fielded. 

There  are  severa.  areas  that  neural  network  technologies  can  help  to 
improve  information  processing,  such  as  parallel  processing,  which  will 
drastically  reduce  processing  time  and  simplify  the  algorithm  development  to 
save  time  and  human  effort. 

2.4.2  Networks 

Neural  network  technologies  will  help  in  the  improvement  of  network 
and  communications  signal  processing,  load  balancing,  and  multi-processing. 
The  Adaptive  Channel  Equalizer  is  now  used  in  virtually  all  long-distance 
telephone  systems  to  stabilize  voice  signals. 
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3.  NEAR  TERM 


3.1  Neural  Network  Capabilities 

It  is  expected  that  within  the  next  seven  years  (up  to  1995)  there  will  be 
many  applications  in  this  field  to  be  fielded  into  military  and  commercial 
world,  when  the  new  hardware  technologies  have  been  developed  to  push  their 
capabilities  beyond  their  present  speed  and  storage  limitations.  While  waiting 
for  the  hardware  development,  many  developers  continue  working  with 
software  emulations  of  neural  networks  that  can  be  used  on  traditional 
computers. 

The  applications  of  Image  Classifier  and  Target  Recognizer  will  help 
the  strategic  defense  in  information  gathering  by  using  electro-optical 
surveillance.  The  applications  of  Sonar  Classifier,  Target  Tracking,  Process 
Monitor,  and  Image  Fusion  will  help  identify  a  target  whether  it  is  friend  or 
foe. 


3.2  Hardware/Performance 

By  the  mid-1990’s,  it  is  predicted  that  a  network  can  be  built  with  10 
billion  interconnects  and  1000  billion  interconnects  per  second.  To  achieve  this 
essential  hardware  requirement,  the  developing  technology  of  Analog  Very 
Large  Scale  Integration  will  mature  to  offer  a  high  density  of  interconnects. 
The  advanced  development  of  GaAs  and  CCD  will  help  to  boost  the  technology 
to  meet  the  expectation.  Last  but  not  least,  optical  technology,  which  has  not 
been  fully  developed,  will  offer  a  very  high  density  of  interconnection  due  to 
its  three-dimensional  nature. 
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3.3  Application  to  IMA 


Without  £.  uoubt  the  developments  in  the  neural  network  field  will  help 
increase  the  power  of  information  processing,  largely  in  the  areas  of  com¬ 
munications  when  transmission  speeds  are  pushed  to  10  billion  bits  per 
second,  and  of  information  when  memory  is  capable  of  handling  100  billion 
bits  of  information.  During  war  time  the  needs  for  data  gathering,  pro¬ 
cessing,  integration,  and  analysis  in  real  time,  or  even  to  offer  suggestions  as 
alternatives  in  decision  making  will  be  urgent  and  vital  to  the  defense. 
Hopefully,  neural  networks  will  provide  these  services  within  the  next  decade. 

4.  LONGTERM 

4.1  Neural  Network  Capabilities 

When  the  development  of  vision,  speech,  signal  processing,  and  sensor 
fusion  technology  is  mature,  the  applications  of  these  technologies  will  be 
integrated  so  that  a  neural  network  can  function  as  close  to  the  brain  as 
possible.  Just  imagine  that  while  a  person  is  driving  a  car,  his  mind  is 
wondering  about  what  he  plans  to  do  for  the  coming  weekend.  In  the 
meantime,  he  is  still  aware  that  he  needs  to  speed  up  to  get  home  for  dinner 
because  of  his  empty  stomach,  and  is  still  aware  that  the  traffic  is  slowing 
down  for  some  reason.  How  to  build  a  machine  to  combine  and  coordinate  all 
the  functions  that  the  brain  can  do  is  still  impossible  until  the  biological 
scientists  understand  how  and  why  the  brain  works. 

4.2  Hardware/Performance 

To  build  a  network  with  100  trillion  processing  elements  and  10.000 
trillion  interconnections  per  second  is  not  impossible  in  the  next  two  decades. 
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To  meet  these  hardware  requirements,  it  will  be  necessary  to  invest  more  in 
the  development  of  wafer  scale  integration  and  optics  technologies.  Can  a 
working  model  of  the  brain  which  functions  as  the  brain  be  built  one  day? 
Whether  such  a  model  will  become  true  or  not  remains  to  be  seen. 

APPENDICES 

A.  GLOSSARY/ACRONYM  UST 

AI  Artificial  Intelligence 
CCD  Charge-Coupled  Device 

DARPA/TTO  Defense  Advanced  Research  Projects  Agency/Tactical 
Technology  Office 
PE  Processing  Element 
SLM  Spatial  Light  Modulator 

B.  DEFINITION  OF  TERMS 

Axon:  In  biological  systems,  the  appendage  of  the  neuron  that  transmits 

electrochemical  signals  away  from  the  cell. 

Cerebral  Cortex:  Major  part  of  the  human  brain  (cerebrum),  the  surface 
layer  of  gray  matter  that  functions  mainly  in  coordination  of  higher  nervous 
activity. 

Dendrites:  Branching  treelike  figure  attached  to  the  ceil,  receive  signals  from 
other  neurons. 

Lavers  for  ” slabs’’^:  Processing  elements  are  arranged  as  layers  in  the  neural 
networks. 

Processing  Element:  The  so-called  artificial  neuron  that  consists  of  several 
resistors  and  operational  amplifiers  and  has  a  small  amount  of  local  memory 
and  processing  power. 
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